Notes on 

Sobolev spaces on compact classical groups 
and Stein— Sahi representations 

Yurii A. Neretin 

We discuss kernels on complact classical groups G = U(n), 0(2n), Sp(n) defined by 
the formula K(z,u) = | det(l — zu*)\ s . We obtain the explicit Plancherel formula for 
these kernels and the interval of positive-definiteness. We also obtain explicit models 
for Sahi's 'unipotent' representations. 

In [15| . the author proposed a class of natural kernels on pseudo-Riemannian 
symmetric spaces G/H and conjectured that these kernels admit explicit Plancherel 
formula. The purpose of these notes is to verify that at least in the case then 
G/H is a compact group GxG/G ~ U(n), 0(2n), Sp(n) this Plancherel formula 
can be written. 1 

0.1. A preliminary example. Recall the construction of unitary represen- 
tations of the complementary series of the group SL(2,R) (Bargmann |IJ, 1947, 
see also 6 ). Consider the unit circle S 1 : z = e lip and the following Hermitian 
form in the space of smooth functions on S 1 



(f,9)s= / |sin(vp/2)| s f(tpi)g((p 2 )d(pidif2. 



o 



This Hermitian form is invariant with respect to the Mobius transformations 
of the circle in the following sence. Consider the group SL(2,R) ~ SU(1,1) 

consisting of all complex 2 x 2-matrices g — ( ^ _) , satisfying \a\ 2 — \b\ 2 = 1. 



Then the operators 



(~ f(z) = f{{a + bz)- 1 (a + bz))\a + bz[ 



l+s 



preserve the form (/, g) s , and moreover this property uniquely determines our 
form. 

For — 1 < s < 1 our Hermitian form is positive definite. This fact follows 
from the identity 

We observe, that for — 1 < s < 1, all the gamma-coefficients are positive, and 
hence (/, f) s is positive for all /. 

1 Of course, the compact groups are the most simple objects among symmetric spaces. Nev- 
ertheless, our calculation (see also 1161 ) is an important heurictic argument for a computability 
of the Plancherel measure in the general case. Indeed, Oshima II 81 recentely observed that 
difference equations for c-function are the same for different real forms of one complex sym- 
metric space Gc/Hc (and hence c-functions of different real forms differs by trigonometric 
factors; actual evaluation of this trigonometric factor is a highly nontrivial result of Oshima). 
It is natural to believe that this phenomenon survives in our situation. 
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It is natural to consider the completion H s of the space C°°(S 1 ) with respect 
to this inner product. Obviously, 

r(n+(l- S )/2), 

a n \ < oo 



<=> E( 1+ H)' 



|a„| 2 < oo. 



The last condition shows that H s is a Sobolev space. 

This construction is the simplest example of a representation that is unitary 
by some nontrivial reason. In fact, Hilbert spaces with various inner product 
having the form 

(f,9)= K(x,y)f(x)g(y)dxdy, 

J JXxX 

where K(x,y) is a distribution, are usual in the representation theory. 

For instance (see Vilenkin, 26,, X.2), conformally invariant inner products 
in spaces of functions on a spere \xi\ 2 + ■ ■ ■ + \x n \ 2 = 1 are given by the formula 



</,<?>= // \\x-y\\- A f(x)g(y)dxdy (0.1) 

and this inner product is positive definite iff < A < n — 1. 

The Hilbert spaces defined by the inner products (0.1) are Sobolev spaces. 
In the representation theory also there are few cases (related to the groups 
U(l, n), Sp(l, n)) when some anisotropic Sobolev spaces arise in a natural way, 
but usually the situation is more complicated (some discussion of functional- 
theoretical phenomena is contained in |17|). 

0.2. Stein kernels. Consider the space Mat„ consisting of n x n complex 
matrices. Consider the following Hermitian form on C°°(Mat n ) 



</,<?>=// \det(x -y)\- 2n+ °f(x)g(y)dxdy (0.2) 

J JMat„ X Mat„ 

(for er > 2n the integral is well defined, further we can consider the meromorphic 
continuation in a). The Hermitian form (0.2) is invariant with respect to the 
transformations 

Pa (l J) f(x) = f((a + xc)- 1 (b + xd))\det(a + xc)\- 2n -°, 

where ^ G GL(2n, C) is an arbitrary invertible (n + n) x [n + n) matrix. 
Stein's theorem. The inner product (0.2) is positive definite iff —1 < a < 

1. 

Vogan [221, 1986, extended Stein's construction to the groups GL(n) over 
real numbers R and quaternions H, see also [201 ■ Sahi [23 > [22] extended the 
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construction to other series of classical groups, precisely to the groups 0(2n, 2n), 
U(n,n), Sp(n,n), Sp(2n,R), SO*(4n), Sp(4n,C), and 0(2n,C), 

The inner products (0.1) and (0.2) seem similar, but the kernel \\x — y\\~ x 
has a singularity on the diagonal x — y; on the contrary | det(cc — y)\~ 2n ~' J has 
a singularity on a complicated surface in Mat„ x Mat„ containing the diagonal. 
In particular, the Hilbert space defined by the Stein inner product (0.2) is not 
a Sobolev space in the standard sence. 

Similarily, various natural integral operators that appear in the representa- 
tion theory usually are not pseudodifferential operators in the standard sence. 

0.3. Sobolev kernels on the orthogonal groups 0(2n). Consider the 
orthogonal group 0(2n), i.e., the group of real 2n x 2n-matrices h satisfying 
h l h = 1. We consider the Hermitian form on C oo (0(2n)) given by 



(F u F 2 )x= \det{u-v)\ A F 1 (u)F 2 (v)d^(u)d^(v), (0.3) 

J JO(2n)xO(2n) 

where /j, is the Haar measure on 0(2n). This form is well defined for A > 0, for 
A < we consider its meromorphic continuation. 

In this paper, we obtain the explicit expression of this form in characters 
(Theorem 3.2) and also show that for — n < A < — n + 1 our form is positive 
definite 

We consider the group 0(2n, 2n) consisting of (2n + 2n) x (2n + 2n)-matrices 

a 

c 



^ I satisfying the condition 



a bV (I OA (a b\ (I 



For an orthogonal matrix h £ 0(2n) and \ a j ) € 0(2n, 2n), we have 



,c d) \0 -1/ \c dj V0 -1 

where the symbol * denotes the transposition of a matrix 

a b 

, c d 

(a + hc^ib + hd) e 0(2n). 

It is easy to verify that the Hermitian form (0.3) is invariant with respect to the 
operators C*°°(0(2n)) C*°°(0(2n)) given by 



p x r b A F(h) = F((a + hc)-\b + hd)) det(a + hc)~ 



2n+l-A 



Thus our Theorem 3.2 gives another proof of Sahi's theorem about existence 
of Stein-type series for the groups 0(2n, 2n). Also, we obtain new models of 
the "unipotent representations" of the groups 0(2n, 2n) (see Sahi, [231 , Dvorsky 
and Sahi, gj, 0). 

0.4. Structure of the paper. We consider the groups U(n), 0(2n), 
Sp(n) in Sections 2, 3, 4 respectively. We also obtain models of Stein-Sahi 
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representations for the groups U(n,n), 0(2n, 2n), Sp(ra,n) In particular, we 
obtain an independent proof of the corresponding Sahi's results. 

In all the cases, we reduce the problem to evaluation of some determinants, 
necessary determinant calculations are collected in preliminary Section 1 (our 
calculations are more uniform than it seems at the first glance, see |16| . 

In Section 5, we discuss models of 'unipotent representations' from 123, 0], 



0.5. Notation. We denote the Haar measure on U(n), SO(n), Sp(n) by /i. 
We assume that the measure of the whole group is 1 . 
The Pochhammer symbol is given by 

/ \ r(a + n) , 

{a) n = a a + 1 ... a + n-1 = \— ' . 0.4 

T(a) 

The second expression also has sence for negative n. 

Acknowledgements. I am grateful to A.Dvorsky for his explanations 
on 'unipotent representations'. This work was done during my visit to the 
Schrodinger Institute, Vienna, winter 2001-2002. I thank the administrators of 
the Institute for their hospitality. 



1. Some determinant identities 

Let A = {aki} be a square n x n matrix, fc, I = 1,2, ... , rt, We denote its 
determinant by det/^; o^;. 

1.1. A determinant of Cauchy type. Recall that the Cauchy determi- 
nant (see, for instance, |TJ) is given by 

i Ui^k<i^n( x k - xi) ■ Ui^kKi^Jyk - yi) n . 

det = — -, . . (1.1) 

kl x k +yi 11 (x fe + yi) 

The following variant of the Cauchy determinant, is also well known. 
Lemma 1.1. 



/ 1 



det 



xi+bi 
1 

xi+b 2 



\ I 

\3;i+f)„_i 



1 

1 

X2+bl 

1 

x 2 +b 2 



X2+&„-l 



1 

1 

1 

X3+b 2 



Xn+bl 
1 

Xn + b 2 



1 I / 

^3+^n-i ' ' ' x n -\-b n -i / 

II i x k + b a ) 

l<a<n-l 



(1.2) 
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Proof. Let A be the Cauchy determinant (1.1). Then 

/ Vl VI Vl \ 

xi+yi \ 

X n +V2 



\x\+y n X2+y-n ' ' ' x n -\-y n f 

We consider lim yiA and substitute y a +\ = b a . 

Vl— >oo 

1.2. One standard determinant. The following determinant is equivalent 
to Lemma 3 from Krattenthaller, 
Lemma 1.2. 



/ xi+yi x 2 +yi 
xi+y-2 X2+yi 



\ 1 



det 



X2+b\ 



(xi+ai)(x 1 +a- 2 ) QE2+ai)(3:2+a2) 
(xi+6i)(a:i+62) (z 2 +f>i)(:E2+£>2) 



n ft+m 

l<m<n-l 



EI (*2+a m ) 

n (^2+6 m ) 

l<m<re-l 



X3+bi 

(a:3+ai)(x3+a2) 

(x3+f>l)03+6 2 ) 



n (i3+«») 

l<m<re-l 



x n +a\ 

Xn+bx 

{x n +a 1 )(x n +a 2 ) 
(x n +b 1 ){x n +b 2 ) 



If (x„+a m ) 

n (^+b^r 

: lsCmsCn-1 , 



II (a*- a;/) II 

l^fc<;n,l</3<n-l 



(a a - bp) 



(1.3) 



Proof. Decomposing a matrix element into the sum of partial fractions, 
we obtain 



(x fc + qQ . . . (x fc + On) 
(x fc + bi) . . . (x k + b a ) 



i 



(«i - bp)(a 2 - bp)... (a a - bp) 

(h - bp)... (6/3—1 - bp)(bp-x - bp) x k + bp 



We observe that the (a + l)-th row is a linear combination of the rows 



xi+bi x 2 +b! 
i X!+b a x 2 +b a 

Thus our determinant is 



n n h (flj " M -det 



/ i 



z„+f>i 



xi+bi K2+&1 



! \ 



Xn+b, 
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and we reduce the evaluation of our determinant to Lemma 1.1. 
1.3. Two determinants. 

Lemma 1.3. Concider the n x n matrix Q having elements 

_ (xk + oyH-Efc + a 2 ) . . . (x fc + a;) _ (x k - ai){x k - a 2 ) ■ ■ ■ (x k - ai) 
(x k + h)(x k +b 2 )... (x k + bi) (x k - bi)(x k - b 2 ) ■ ■ ■ (x k - h) ' 

Then 

2" n x k - n n (&«+m- n k-m 
detg = n — K^m ' 

1 ;=C fc ;=C n , 1 a ^ n— 1 

Proof. We expand a matrix element in a sum of partial fractions. 
We write 

11 2x k 



x k + b a x k - b a x\ - b 2 a 
and observe that an Z-th row is a linear combination of vector-rows 



( Xl x k \ 

\ x'f-b'j ■ ■ ■ xi-bi )> 



Thus our determinant is 

A nU 



and the last factor is reduced to the classical Cauchy determinant (1.1). 
Lemma 1.4. 

dct | ^ + -L)1^ + Q2 ) • ■ ■ + a ') + ( Xfe ~ a i)( a; fe - «2) • ■ • (x k - oj) 



fe,i I (xk + b l ){x k + 6 2 ) . . . (x fe + h) (x k - h)(x k -b 2 )... (x k - bi) 

2 n (xi-xf)- n (6a+6/j)- n k-m 

l^fc<Z^n l^a^/3^n-l l^a^/3<n-l 

n R - ^) ' 

1 < fc < n . 1 < a ^n— 1 



(1.4) 



Remark. In particular the first row of our matrix is (2 ... 2). 
Proof. Decomposing a matrix element into a sum of prime fractions, we 
obtain 

y n°=i(Qj-fe«) / 1 1 
9feZ sn"=i(6i-6a)^*+ & i 



G 



Since, 



-2b k 



x k +bi Xk — h x 2 k -bf' 
the Z-th row is a linear combination of the vectors-rows 



bi 



( J"- 1 b -z. \ 

V x 'i- b 't-i "' x l-K-i ) 



and we obtain 



2 TT ?h TT {ai ~ h)(a2 - h) . . . (ai - b t ) 

i<-< i J i</< i (6i - 6i)(&2 - 6i) ■ - ■ (6»-i - 60 ' 



x det 



( 1 1 

i i 



2 °1 



i i 



1 \ 



It remains to apply Lemma 1.1. 



2. Sobolev kernel on the unitary group. 

2.1. Definition of kernel. Let z be an n x n matrix with norm < 1. For 
a G C, we define the function det(l — z) a by 



dct(l - z) 17 := det 



C7((7-l) 2 <7(<7-l)((7-2) 3 

1 - + — ^ — -g 2 -g A + ... 



2! * 3! 
We also define this function for z satisfying ||^|| = 1, det(l — z) ^ being 

det(l-z) CT ~ lim det(l-u)' T . 

u^z. |u||<l 

The expression det(l — z) a is continuous on the domain ||z|| ^ 1 except the 
surface det(l — z) = 0. 

We denote by det(l — z)^ a ^ the function 

det(l - z) {ct|t} := dct(l - z) a dct(l - z) T . 

We define the function l<j, T (g) on the unitary group U(n) by 

£ a T (g) : = 2-( <T+r )"dct(l - z){ ff l r >. 

Obviously 



C ;T (/i V) = C,r(.9) for .9, /i e U(n). 



(2.1) 



7 



Lemma 2.1. Let e 1 ^ 1 , e 1 ^™ be the eigenvalues of g € U(n); we assume 
^ ipk < 27r. TTien 



%) - exp{( ( r - r) - tt)/2} J] sin 



fe=l 



Proof. It is sufficient to verify this statement for diagonal matrices, or 
cquivalenly we can check the identity 



We have 



i(l- e ^)=exp{i(^-7r)/2}si n |. 
Further, in the equality 

2-^(1 _ e^) 17 = exp{ia(il> - tt)/2} sin' 7 |, 

the both sides are real-analytic on (0, 2ir) and the substitution ip = it gives 1 in 
the both sides. □ 

We also define the kernel L<j tT (g, h) on U(n) by 

LaA9-h) = LAah- 1 ). (2.2) 

Obviously, this kernel is invariant with respect to left and right shifts on U(n), 
i.e., 

La,r(rigr2,r 1 hr 2 ) = L a ^ T (g,h) for g, h, r x , r 2 £ XJ(n). 

2.2. Characters, see Weyl book j2Hj- The set of finite dimensional repre- 
sentations of U(n) is parametrized by collections of integers (signatures) 

m : mi > TO2 > • • • > m„. 

The character \m of representation 7r m (a Schur function) corresponding to a 
signature m is given by 

det fc7 --i2 Je m ^ k \ , s 

det fcj= i i 2,...,n{eH J 1 ^ h ) 

where e l ^ k is the eigenvalues of g. Recall that the denominator admits decom- 
position 

JJ(e** - e**»). (2.4) 

Kk 

The dimension of 7r m is 

m IIo<a</3<n( m ° ~ mg) 

dmi7r m = Xm(l) = ^pi r. ■ (2.5) 

1 lj=i J ■ 
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A function F(g) on U(n) is central if it satisfies the identity F(h 1 gh) = 

F{g). 

Consider the Haar measure \i on U(n) normalized by the condition: the 
measure of the whole group is 1. For a central function on U(n), the following 
Weyl integration formula holds 

F(g) d»(g) = 

m / ■ • • / ^(diag(e*, . . .,e*»))| [J ( e ^-e*)| 2 f[ 



U(n) 



(27r)"n 

0<i/>i<27r 0<-!/'ti<27t 



m<k k—1 

(2.6) 



where diag(-) is a diagonal matrix with given entries. 

A central function F £ L 2 (U(n)) admits an expansion in characters, 



F(g) = E 



where the summation is given over all the signatures m and the coefficients c n 
are the L 2 -inner products 



c m = / F(g)xm(g)dfi{g). 

JU(n) 

Applying formula (2.6), explicit expression (2.3) for characters, and formula 
(2.4) for the denominator, we obtain 

c m = ^ / ... / F(diag{e^,...,e^})x 

O<-01<27T 0<V'n.<27T 

n 

x det { e *CJ-i)**\ det | e -«^<M TT d Vk . (2.7) 

fc,j=l,2,...,n 1 J fc,j=l,2,...,n L J J. J. r 

fc=l 

For calculation of such expressions, we will use the following evident Lemma 
(see, for instance, [T5|.1 

Lemma 2.2. Lei X be a set, 



TT /(zfc) det {uj(a;fc)} det {^(xfc)} TT 

n fe,i=l,...ri l,...n A A 



J 



i! det 

Lm— 1,.. 



/(x)ui(ar)v m (x)dai| 
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2.3. Lobachevsky beta-integrals. We will use two following integrals, 
see 0, 3.631,1, 3.631,8, 

' . \, s „ s , 2 1 -^7iT(V)cos(&7r/2) 

sm'" 1 ?) cos ta; da; = — r- V /, s , s 2.9) 

k ^ ' r((/x + 6 + l)/2)r((/i-6 + l)/2) 1 ; 

In some sence, our integral evaluations below, 2.4, 3.3, 4.3, are multivariate 
analogs of these integrals. 

2.4. Expansion of the function i a , T - 
Theorem 2.3. Let Re(<r + t) < 1. TTien 



_ (-l) n ( n - 1 )/ 2 sin n (7ra)2-( CT+T )' 



n ( m "- TO /3)ii 



3=1 

r-r r(— CT + TOj — 71+1) 



(2.10) 

n 

= ( _ 1 )n(r l -l)/2 2 -(<x+r)n TT r((T + T + j) X 
3=1 

(_1)E^ J] (ma-m^) , 

x £ -S ■ 

m I J] r(<r - mj r + n)r(r + mj + 1) J 
3=1 

Proof. We must evaluate the inner product 



'U(n) 

Applying (2.7), we obtain 



/ ZaA9) Xm{g) d(i,(g) 
JV!n) 



(27T) 



1 p n 

J J] [sin^(^/2) • exp{i(a - r){^ - tt)/2} 



x det {e"™^'}- det {e^- 1 ^ } TT <f^. 

l<fe, l<n l<k,l<n 1L 

1=1 



By Lemma 2.2, we reduce this integral to 

1 



(271-)™ i<fe,jsSn 



det J(fc,i), 
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where 

/■2tt 

I(k,j) = e- i ^- T ^/ 2 / sin' T+T (^/2) -exp{i((CT + T)/2 + fc- 1 - m^} dep. 



o 



We apply (2.8) and obtain 

2 1 - a - T 7rT(a + r + 1) (-l)^ 1- 



I(k,j) 



r(cr + k- m,j)T(T - k + mj + 2) 
Applying standard formulae for L-function, we obtain 



J(fc, j) = 2 1 - <T - T r(cr + t + 1) sin(-cr7r) 

1 -f TOj 

{—a + mj — n + l) r 



= 2 i - ff - r r(CT+T+l) sin(-erTr) 



T(r + rrij - k + 2) 
r(-cr + mj - n + 1) 



r(r + mj- - n + 2) 



(t + mj - n + 2)„_ fc 



The factors outside the box do not depend on on k. Thus, we must evaluate 
the determinant 

(-0- + m 3 - n + l)„-fc 
(t + rrij — n + 2)„_fc 

Up to a permutation of rows, it is a determinant of the form described in Lemma 
1.2 with 

Xj = rrij, aj = —a — n + j, t = —n + j + 1. 

After a simple rearangemcnt of the factors, we obtain the required result □ 
2.5. Hermitians forms defined by kernels. First, recall some standard 

facts on characters of compact groups, for details see, for instance, [Hj, 9.2, 11.1. 
Let K be a compact Lie group equipped with the Haar measure fj,, we assume 

that the measure of the whole group is 1. Let nx, TT2, ...be the complete 

collection of pairwise distinct irreducible representations. Let xi> X2, ■ ■ ■ be 

their characters. Recall the orthogonality relations 



(X*,XI>L»:=/ Xk(h) X i(h)dn(h) = S k ,i (2.12) 
Jk 

and 

{-r-i — Yi., if k = 1 , 

r (2 - i3) 

where * denotes the convolution on the group, u * v(g) — J u(gh~ 1 )v(h)d[i(h). 

We consider the action of the group K x K on K by the left and right shifts 
(fci, k%) '■ g ► k\gk2- The representation K x K in L 2 (K) is a multiplicity free 
direct sum of irreducible represntations having the form 7T& <£> 7r^, where 7ij! is 
the dual representation, 

L 2 {K)^($p k ®p* k . (2.14) 
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Each distribution / on if is a sum of "elementary harmonics" 

The summands of this sum correspond to the decomposition (2.14). 

The projector to a subspace pk ® p% of fc-th elementary harmonics is the 
convolution with the corresponding character, 

f {k} = ^~ f*Xk (2.15) 
dim n k 

(in particular, is smooth). 

For a central distribution S on K, consider the Hermitian form 

(u, v) = E(h, g) u(h)v(g) dfj,(u) dn(g). 

J JKxK 

Consider the expansion of 5 in characters 

k 

Lemma 2.4. 

(u,u)=V— ^— / uW(h)vW(h)dfi(h). (2.16) 
V dimTTfe 7 u(n) 

Proof. We can assume u = Xk, v — xi- We evaluate 

/ / y^CkEigh-^XkWx^d^dfiih). 

JJKxK k 

using (2.12) and (2.13). □ 
2.6. Positivity. Let Re(cr + r) < 1. Consider the sesquilinear form on 
C°°(U(n)) given by 

(q,r) a ,r= L <J ^ T (g,h)q(g) : F{h)dgdh, (2.17) 

JJU(n)xU(n) 

where the kernel L a , ta u is the same as above. Obviously, for fixed q, r, this 
expression admits a meromorphic continuation in a, r to the whole C 2 . More- 
over, Theorem 2.3 allows to write an explicit expression for this continuation. 
Expanding q and r in elementary harmonics 

q(h) = J2*™(h), r(h) = J2^(h), 

m m 

we obtain (see Lemma 2.4) 

(q,r)cr,T = V] -jr-^ — / >c m (h)9 m (h) dfx(h), 
m clim7r m J u(n) 
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where the meromorphic expressions for c m were obtained in Theorem 2.3. 

If a, t € R, then our kernel L CT . T is Hermitian, i.e., La- tT (h,g) = L aiT (g,h), 
or eguivalently 

{q,r)„ tT = (r,q) aiT 

Corollary 2.5. For a,T G R\Z, the inner product (2.17) is positive 
definite (up to a sign), iff fractional parts of —a — n and t are equal. 

The domain of positivity is the union of the dotted squares on Figure 1 . 
For a, t satisfying this corollary, denote by H a ^ T the completion of C°°(U(n)) 
with respect to our inner product. 

2.7. Action of U(n, n) on the space U(n). Consider the linear space 
C™ © C™ equipped with the indefinite Hermitian form 

{v © w, v' © w'} = (v, t/)o©o - (w, «/)o©o 

where (•, •) is the standard inner product in C™. Denote by U(n,n) the group 
of linear operators in C™ © C™ preserving the form {•, •}. We write elements of 

this group as block (n + n) x (n + n) matrices g := ^ . By definition, 

these matrices satisfy the condition 




(2.18) 



For h € U(n), consider its graph graph(ft-) in C"©C". It is an n-dimensional 
linear subspace, consisting of all vectors z © zh, where a vector-row z ranges 
in C n . Since h € U(n), the subspace graph(/i) is isotropic 2 with respect to 
our Hermitian form {•,•}. Conversely, any n-dimensional isotropic subspace in 
C™ © C™ is a graph of a unitary operator fteUfn). 

Thus we have one-to-one correspondence between the group U(n) and the 
Grassmannian of n-dimensional isotropic subspaces in C" © C™. 

The group U(n, n) acts on the Grassmannian in an obvious way, and hence 
U(n, n) acts on the space U(n). An explicit formula for the latter action can be 
easily written: 

9=(^ fj ■ h^hW := (a + zc)- 1 {b + zd), h G U(n), g G U(n, n) 

(2.19) 

LEMMA 2.6. a) For the Haar measure ^i(h) on U(n), we have 

Li(h [g] ) = \dct- 2n (a + Z1 )\-n(h) 
b) The kernel L a ^ T satisfies the identity 

L a , T {u\ g \ v^) = L ct>t (m, v) det(a + m 7 )^I t > det(a + vj)^ a "> . (2.20) 

2 A subspace V in a linear space is isotropic with respect to an Hermitian (or bilinear) form 
Q if Q equals on V. 
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Proof, a) The differential of the map u i— > is given by 

duM (a + u-/)- 1 du (~-/u [g] + 5), (2.21) 

see, for instance, ^3], Lemma 1.1. 

The rational map u v-t is defined on the space of all complex n x n 
matrices. By (2.21), its complex Jacobian is 

J(g, u) := det~"(a + zc)- n dei n {- lu ^ + 5). 

Hence the real Jacobian on the space of all matrices is |J(<7,w)| 2 , and the Ja- 
cobian of the map U(n) — > U(n) is \ J(g,u)\. It can easily be checked (see |13) . 
Lemma 1.2) that 

det(-7u [ffl +S) = det(a + 27)" 1 det(g). 

By (2.18), I det g\ — 1, and this finishes proof, 
b) A direct calculation. 

2.8. Unitary representations of U(n, n). Let U(n, n) acts in the space 
of C°° -functions on U(n) by the operators 

p a . T {g)F{h) = F(h [9] )det { - n ~^- n ~ T} {a + hj). (2.22) 

Remark. Let us explain the sence of the complex power in this formula. It 
can easily be checked with (2.18), that 1 1 'yo; 1 1 1 < 1. Hence, for all matrices h 
satisfying \\h\\ ^ 1, the matrix a + /17 is invertible. Hence the function 

In det (a + /17) 

has a countable family of continuous branches on the set \\h\\ ^ 1 and in partic- 
ular on U(n). We define 

det { - n - al ~ n - T} (a + hj) := exp{-(n + a) In dot (a + fry)- 

- (n + r)lndct(a + hrf) - 2mk(<r - r)}. (2.23) 

Thus, we can think that for each g € U(n, n) formula (2.22) defines a countable 
family of operators Pa, T {g)i they differs one from another by constant factors 
exp{27rifc(er — r)}. These operators define a projective representation (see Hj,14) 
of the group U(n, n) 

Pa,r(g)pa,r(g') = X(g, g') Pv, T {gg' ') , X(g, g') G C. 

Equivalcntly, wc can consider the multi- valued operator- valued function p a>T (g) 
on U(n, n) as a single-valued function on the universal covering group U(n, »)~ 
of U(n, n). Then p^^ig) became a linear representation of U(n, 

If (a— r) € Z, then exp{27rifc(tr— r)} and (2.23) is a well defined single- valued 
expression. In this case p aiT is a linear representation of U(n,n). 
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Proposition 2.7. The operators Pa, T {g) preserve the form (•,-)a lT . 

Proof. First, let Rc(ct + t) < 1. Substitute hi = v)?\ h 2 = vJf to the 
integral 

/ / L CTjT (/ii, h 2 ) q(hi)r(h 2 ) dp,{h\) dp(h 2 ). 

J iu(n)xU(n) 

By Lemma 2.5, we obtain 

/ / L CT;T (ui,u 2 ) det(a + ui7) { ~ CT '~ r} | dot (a + M 2 7)|~ r| ~ CT} x 

JJU(n)xU(n) 

x q{h-t)r{h 2 )\ det(a + ui7)|~ 2n | det(a + u 2 7)r 2 ™ rf^(ui) d/z(u 2 ). 

Thus, our operators preserve the form (•, -) a ,T- 

For general <r, r, we consider the analytic continuation. 

COROLLARY 2.8. For a, t satisfying the positivity conditions of Corollary 
2.5, the representation p a ^ is unitary. 

2.9. Some remarks, a) The case n = 1, a = t = s — 1 gives precisely the 
complementary series of representations of SL(2,R) ~ SU(1, 1) described above 
in 0.1. 

b) The representations p„ tT are very degenerated in the followng sense. For 
any irreducible unitary representation of a scmisimple group G, its restriction 
to the maximal compact subgroup K has a spectrum with finite multiplicities, 
but usually these multiplicities are not bounded. 

In our case, i.e., G = U(n, n), K — U(n) x U(n), the restriction of p CT>r to K 
is the multiplicity free sum p m <g> p* m (thus, only few representations of K are 
present in the spectrum). 

c) Shifts of parameters. For integer k, the projective representations p a +k.r-k 
and pa- tT are equivalent. The intertwining operator is the multiplication by the 
determinant 

F(h) ^ F(h)det(h) k . 

This operator also defines an isometry of forms L a+ k, T -k and L CT , T 

d) Symmetry. Representations p_„/ 2 _ Pi _„/ 2 _ g and P- n / 2 + P ,-n/2+q are dual. 
The invariant pairing is given by the formula 

(F U F 2 )^ f F 1 {h)F 2 {h)dn{h). (2.24) 

JU(n) 

For verification of this statement, we substitute h i— » hfo and apply the 
formula for the Jacobian. 

In particular, the point (<r, r) = (n/2, n/2) corresponds to a representation 
of a unitary principle series of U(n, n) (it is unitary in the space L 2 (\J(n))). 
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Figure. Groups U(n). The case n = 5. 

1. The dotted squares correspond to unitary representations p a . T . 

2. Vertical and horizontal rays in the south-west of Figure correspond to nonde- 
generated highest weight and lowest weight representations. Fat points correspond 
to degenerated highest and lowest weight, representations, and also to unipotent 
representations. The point (a, r) = (0,0) corresponds to the trivial one-dimensional 
representation. 

3. In points of the thick segments, we have some exotic unitary sub-factors. 

4. The shift (<r, r) {a + l,r— 1) transform a projective representation p„ tT 
to an equivalent representation. 

5. The permutation of the axes (r, a) (ct, t) gives a complex conjugate 
representation. 

6. The symmetry with respect to the point (— n/2, — n/2) (black circle) gives a 
dual representation (for odd n this point is a center of a dotted square; for even n 
this point is a common vertex of two dotted squares). 

7. For a + t = n (the skew diagonal line) our Hermitian form is the standard 
i 2 -product. The represenations p ajT in this case are in the untary principal series. 

8. Linear (non-projective) reprsentations of U(n, n) correspond to the family of 
parallel lines c-reZ. 
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e) Another symmetry. The representation p TIJ is complex cojugate to p a ^ r 

f) Problem of unitarisability of subquotients. Corollary 2.8 gives a classi- 
fication of unitary representations among p a ,r- But for integer a or integer 
r, the representation p a ^ can contain a unitary subrepresentation, a unitary 
factor-representation, or a unitary sub-factor. 

g) Unitary highest weight representations and Sahi's unipotent representa- 
tions. The kernels 

La-,o(u, v) — det CT (l — 

are well-known, and they define highest weight representations. By a well- 
known theorem (Berezin, Gindikin, Rossi- Vergne, Wallach), the form L a fi is a 
nonnegative Hermitian form iff 

a<-n+l and for a = -n+ 1, -n + 2, 0. (2.25) 

First, let a < — (n— 1), If m n is negative, then the factor r(0 + mj ■ + 1) in the 
denominator of (2.10) is infinity, and hence (2.10) is 0. If m n ^ 0, then all other 
coeffitients c m have the same sign (all signs are positive or all are negative). It 
is easy to observe this from (2.10) for noninteger ex. For integer a, this follows 
from the limit considerations: numerator and denominator in (2.19) have poles 
of the same order, the limit as a — * — I have to be nonnegative as a limit of a 
nonnegative function. 

Second, let a be in the discrete part of the set (2.25), a = —n + 9. Then the 
factor Y[ r(a + j) has a pole of order n — j. The factor Y[ T(cr + n — mj) has a 
pole of order ^ n — j (since m n 0, m n _i 1, . . . , mi ^ n — 1). Hence the 
ratio is finite, it is nonvanishing if the order of a pole of denominator is precisely 
n — j. This happens iff m n = 0, mi = 1,. . . , m n -j + i = j — 1. After this, it 
remains to follow signs in (2.11). 

Remark. This consideration almost coincides with the original Berezin's 
proof [2] • Hua Loo Keng [Sj have obtained an expansion of the kernel L&,o in 
a series of characters (it is a partial case] of our Theorem 2.3), and Berezin 
checked signs in this expansion. 

In integer points lying in the strip ^ a + t ^ — n + 1 also there are located 
unipotent representations I2S1, see below Section 5. 

h) Some other unitary subquotients. Consider a representation p a T lying on 
the boundary of the dotted domain on Fig.l. For definiteness, assume r = 0. 
Obviously, each term of its Jordan-Holder series is unitarizable (since a limit of 
positive inner products is positive). Denote by Yj the set of all the signatures 
m satisfying the condition 

m n -j > 0, m n -j+i < 

(i.e., precisely j terms of the signature are negarive). For m £ Yj, the expression 
c m (2.19) has a zero of order j. 

Denote by Wj, the subspace in C°°(U(n)) spanned by all harmonics with 
signatures lying in Uk^jYj. We obtain an invariant filtration 

C°°(U(n)) = Wo D Wi D ■ ■ ■ D W n 
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The representation of U(n, n) in each sub-factor Wj/Wj+i is unitaty. 
i) Matrix Sobolev spaces of an arbitrary order. Denote 



Let F be a distribution on U(n), let F = ^ F m be its expansion in a series 
of elementary harmonics. We have 

FEH a , T <=► V^^||F m || 2 L2 <co <=► 
^ dim 7r m 

n 

m 

where ||-F m ||L 2 denotes 

\\F m \\ L *:=U \F m (h)\ 2 dn{h))' 



3=1 



1/2 



U(n) 



Our Hermitian form defines a norm only in the case |s| < 1, but (2.26) has sence 
for arbitrary real s, and thus we have a possibility to define a Sobolev space IH S 
on U(n) of an arbitrary order. 

Author do not know applications of this remark, but it seems that it can be 
useful in two following situations. 

First, a reasonable harmonic analysis related to semisimple Lie groups is 
the analysis of unitary represenations. But near 1980 Molchanov observed that 
there are many identities with special function that admits interpretations on 
"physical level of rigor" as formulae of nonunitary harmonic analysis. Up to now, 
there are no reasonable interpretations of this phenomenon (see, for instance [5], 
see also [T2|, Section 1-32 and formula (2.6)-(2.15) ). In particular, we do not 
know reasonable functional spaces that can be place of action of this analysis. 
It seems that our spaces H s can be possible candidates. 

Second, natural integral operators in the noncomutative harmonic anal- 
ysis seem similar to pseudo-differential operators, but they are not pseudo- 
differential operators in the usual sence. In particular, they are not well com- 
patible with the standard scales of functional spaces. It can happened that our 
spaces !K S can be more reasonable in this situation. 

j) An identity for formal series. We have (see notation (0.4)) 



|sin(<£/2) 



a+T (a _ T)tp = 2^r(q + T+l) (~1)"V"^ 



Introduce new variables x p — e %Vp . We can rewrite Theorem 2.3 as the following 
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identity for formal series 



n 

n{ 



(-i) m 4 



^ (r+l) ro (o- + l)_ T 



n 

E[n 



(-1)* 



=i ( T + % ( CT + n )-^ J {deti^, p<n 4 1 } 



{deti^^p^^ Xp } 



3. Orthogonal groups 

3.1. Definition of the kernel. We consider the (disconnected) group 
0(2n) as a basic object 3 . Each clement of this group can be reduced by a 
conjugation g i— ► hgh^ 1 to the block diagonal form with 2 x 2-blocks 



o ... o \ 

A(cp 2 ) ... 



V ... Afo>„)/ 



, where A( Vj ) = ( ^ SmtPj ). (3.1) 
< — sin <f j cos ip j J 



The collection (ifi, . . . ,tp n ) is uniquely determined by g modulo permuta- 
tions and arbitrary transformations <pj >— > —<Pj. The numbers e^ tVj coincide 
with the eigenvalues of g. 

We define the function &\{g) on 0(2n) by 



= fdet((l-<?)/2) A , ,if 5 eSO(2n); 
}0, if 5 ^SO(2n). 



the derminant is nonegative and hence its complex powers are well-defined. In 
the terms of the eigenvalues e ±lipj , the function £\(g) coincides with 

n 

£ A (p) = rj|sin(^/2)| 2A . 

i=i 

We define the kernel L\(-, •) on 0(2n) as 

L x (g,h)=£ x (gh- 1 ). 

3.2. Characters, see [21]. Irreducible representations tt\ of SO(2n) are 
parametrized by collections of numbers 

(l,e): h > h > ■■■ > In > 0, e = ±l, 



3 It is also possible to concider its connectd subgroup SO(2n); in this case we must replace 
the group 0(2n, 2n) below by it connected component SOo(2n,2n), consider a connected 
component of the Grassmannian and also do some obvious minor changes. 
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or 

1 : h > h > ■ ■ ■ > L = 0. 

Formulae for the characters are slightly different in these two cases. 
If l n = 0, we have 

_ deti^fc {cos(7 fe <^ m )} 

deti^fc {C0s(fc - l)ip m } ' 

For Z„ ^ 0, we have 

deti^ msS „{cos(7fc¥>m)} +£deti^ fe , msS „{sin(^^ m )} 



2deti^ msS „{cos(fc - l)ip m } 



Denote by J the diagonal matrix <G 0(2n) having (2n — 1) entries (+1) 
and one (—1). The map h i— » JhJ^ 1 is an interior automorphism of 0(2n) 
and an exterior automorphism of SO(2n). The representations p, of SO(2n) 
corresponding to the characters xt are twins in the following sence 

irfiJhJ- 1 ) = n?(h). 

Also the substitution 

changes xt an d X\ ■ 

For l n = 0, the representation tt\ is stable with respect to the exterior auto- 
morphism h i ► JhJ^ 1 . 

This digression about two types of characters is almost non-essential for us, 
since really we will consider arbitrary 

1 : h > h > ■ ■ ■ > In > 0, 
and the functions xi given by 

deti <fc;m ^»{cos(/ fc ^m)} _ |xi(ff), if L = 0, 2 ^ 

det i! gfc im ^„{cos(fc - l)<^ m } (sO + X\(g), iH„ > 0. 

Remark. The functions (3.2) are precisely restrictions of the characters of 
the disconnected group 0(2n) to the connected group SO(2n). 

Let F be a central functions on SO(2n). Let / be its restriction to the 
maximal torus, i.e., to the set of matrices having the form (3.1). The Weyl 
integration formula for central functions on SO(2n) has the form 

J F(g)dn(g) = 

SO(2n) 

= J J /(^>---^n)( i< det^{cos(fc-lV m }) 2 ^ 1 ...#„. 

0<i/>i<2tt O<-0„<27T 
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3.3. Expansion of i\ in characters. 

Theorem 3.2. Let A < 1/2. For g e SO(2n), 



hl n =0 



i: ;„>o 



where 



Cl = (_l)n(n-l)/2 2 2nA+l 7r -n sm ™ TrA ]J T(2A + 2fc - 1) X 

r(/j - A-n + 1) 



fc=i 



n di-^-u 



r(Z 3 - + A + n) 



(3.3) 



= (_ 1 jn( n -l)/2 2 2nA+l jj r ( 2 A + 2fc - l)x 

„ (-i) EI 'ni^fi-i?) 



fe=l 



nLrHi + A + n)r(i j + A + n)) 



• (3-4) 



Proof. First, e x (JhJ- r ) = £ x (h). Hence 



Evaluating the last expression, we obtain 
1 



TT n n\ 



J ... J n|sin(^/2)| 2A x 

~. ^Ott n^,~ K=l 



0<ipi<27r 0<ip n <2n 



x dct {cosZj^fe} det {cos(j — l)(fk} TT cfysfc. 



fc=l 



Applying Lemma 2.2, we transform this integral to 

■K~ n detl(j, m), 

where 

/* 27r | |2A 

I(j,m) = / sin(<£>/2) cos(lj<p) cos[(m — l)y>] dy>. 
Jo 

Expanding the product of cosines, we obtain a sum of two integrals of the form 
(2.9) 

/t"7T 

1 2A r i 

sirn^l cos 2(1, + m — l)ip + cos 2(1,- — m + l)ip dip 



Jo 
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7r2~ 2A r(2A+ 1) 



r(A 



m)r(A 



2) 



+ 



(-1) 



lj +m— 1 



r(A + lj - m + 2)r(A - lj + to) 



After simple transformations, we get 

T(-l- X + lj+m) r(l-A + Zj -m) 
+ r(2 + A + lj - m) 



2" 2A r(2A + l)sinA7r 
T(2A + 1) sin(ATr) T(lj - A) 



2 2A 



r(ii + a + 1) 



(lj - A) m _i 



(Zj + A — m + 2) m _j 



-(Zj + A + l) m _i (Zj - A - m+ l) m _i 



The factors outside the boxed equation do not depend on to. Thus, we must 
evaluate 



det 



(Z, - A) r . 



+ 



(lj + A - m + 2) m _i 



.(Zj+A+l) TO _i (Zj - A - to + l) m _i 
A matrix element can be represented in the from 



(l j -X)(l j -X + l)...(l j -X + m-2) 
(lj + X + l)(Zj + A + 2) . . . (lj + X + m - ] 



+ 



(lj + A)(Zj- + A - 1) . . . (lj + A - to + 2) 
(Zj - A - — A — 2) ... (Zj — A — to + 1) 



(in each numerator and denominator we have (to — 1) factor; in particular for 
to = 1 we have hji = 2). Thus we obtain a determinat evaluated in Lemma 1.4 
with 

iEj = Zj, aj = —A + j — 1, bj = Xj + j. 

After some rearangement of the factors we obtain the required result. 

3.4. Representations. Now we can repreate the considerations of 2.6-2.8. 
Consider the linear space R 2n R 2 ™ equipped with the indefinite bilinear form 

{v 8 w, v' 8 w'} = (v, v') R2nm - (w, u/) 0ffiR 2„ , 

where (-, ■) is the standard inner product in R 2n . Denote by 0(2n, 2n) the group 
of linear operators in R 2 ™ © M. 2n preserving this form. 

For h e 0(2n) consider its graph graph(/i) in R 2 ™ © R 2 ™. Since h £ 0(2n), 
the subspace graph(/i) is isotropic with respect to our bilinear form {•,•}. 

Thus we have one-to-one correspondence between the group 0(2n) and the 
Grassmannian of 2n-dimensional isotropic subspaces in R 2 ™ © R 2 ™. 

The group 0(2n, 2n) acts on the Grassmannian and hence 0(2n, 2n) acts 
on the space 0(2n). The explicit formula for the latter action is the same as 
above, see (2.19). 

We consider the inner product in C°°(0(2n)) given by 



II 



0(2n)xO(2n) 



Lxihuhi) FifH) F(h 2 ) dnih^dfifa). (3.5) 
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This integral is convergent if A > —1/2, for general A we consider the analytic 
continuation. 

Proposition 3.2. a) The inner product (3.5) is invariant with respect to 
the transformations 

P x(g)F(h) = F(h^)det(a + hj)- 2n+1 -\ 

where g £ 0(2n, In), h £ 0(2n). 

b) If — (n — 1) > A > —n, then the inner product (•, -)\ is positive definite. 
In other words, the representation p\ in this case is unitary. 

Proof. Statement b) follows from Theorem 3.1. For evaluation of the 
Jacobian, see, for instance, [T3j, 1.2-1.3. 

4. Symplectic groups 

4.1. Quaternionic matrices. We denote by H the quaternionic field. 
Operators in the quaternionic coordinate space H n can written in the form 

v i — ► vQ, 

where Q is an n x n-matrix with quaternionic elements, and v £ H" is a 
vector-row. More formaly, these transformations are endomorphisms of a left 
n-dimensional module over the field H. 

Let Q be a quaternionic matrix. It defines the operator H" — > H™, identifying 
H" with R 4 ™, Hence we obtain an operator Q R : R 4 ™ M 4 ". We define the 
determinant of Q by 

detQ = v' detQR. 

(det Qr is a nonnegative real number, hence det Q also is a positive real number) . 
A quaterninic matrix Q is unitary, if it preserves the Hermitian form 

{v,w} = ^v j w J . 

In other words 

QQ* = 1, 

where Q* is a matrix obtained from Q by transposition and element-wise quater- 
nionic conjugation; if qki are the matrix elements of Q, then the matrix elements 
of Q* &req lk . 

4.2. Symplectic group, see j^Hl- The group of all n x n quaterninic 
unitary matrices is called "symplectic group" or "compact symplectic group", 
the standard notation is Sp(n). 

Using a group conjugation h i— > g~ 1 hg, each element of Sp(n) can be reduced 
to a diagonal matrix with the entries e lkv . The collection ipj is defined uniquely 
modulo permutations and change of signs: ipj i— > (— l^tpj. 
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Let F be a central function on Sp(n), let / be its restriction to the set of 
diagonal matrices. The following Weyl integration formula holds 

f F{g)dfi{g) = [ ... [ f(ip u ...,ip n )x 

JSu(n) T n U\ J J 



'Sp(n) 

O<-01<27T 0<1/J„<27T 



i2 " 

det {sin(fcVj)} ]\ d ^3- 

Finite-dimensional irreducible representations of Sp(2n) are enumerated by 
collections 1 of integers having the form 

1 : h > l 2 >•••>/„> 0. 

The corresponding character is 

^ ^ _ deti^j,fc<g„{sm(l fc ^ j )} 
deti^ ifc ^ n {sin(fc^j)} 

We define the function l\ on Sp(n) by 

n 

£ x (g) = det((l - 5 )/2) 2A = J] | sin(^-/2)| 2 \ 

4.3. Expansion of /a- 

Theorem 4.1. for Re A < 1/2, 

l 

where 

n 

c, = 7r-"2- 2nA sin" Att JJ T(2A + 2fc)x 
fe=i 

xfi(2u n ^-^n r a ( +A+i+ fc ) - (4 - i} 

a=l 0<a</J<n j=l 1 l«.7 + ^ + 1 + «J 

Proof is very similar to the orthogonal case. We must evaluate the deter- 
minat whose matrix elements are 

I 2A 

I(k,j):= / sin(y>/2) sin(ljip) sm(kip) dip. 
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Expanding the product of sines, we obtain a sum of two integrals of the form 
(2.9) 



Jo 



2A 

sin(<p/2) [cos2(/j — k)ip — cos2(/j + k)ip] dip = 
= 7r2~ 2A r(2A + l)x 



r(/j + a - k + i)r(-Zj + a + fe + 1) v{ij + a + k + i)r(-z j + a - fc + 1) 



7r2- 2A r(2A + l) 



r(ij - a - fc) r(ij- - a + *;) 



.r^ + A-fc + i) r(/j + A + fc + i) 

We transform the expression in the brackets to 

r fe-A) r (ij + A - fc + i) fc (/j-A) fc 



— }• 

1)J 



r^ + A) i {ij-x-k) k {ij + \ + i) k 

The factor in the front of brackets does not depend on k and hence it is suffitient 
to evaluate the determinant of the matrix 

= (lj + A — fc + l)fc _ (lj - A) fc 
Uk] (lj - A - k) k {I, + A) fc ' 

This matrix has the form described in Lemma 1.3, with 

Xj = lj, at = A + 1 — k, bk = —A — k. 

4.4. Unitary representations of Sp(n,n). The pseudounitary quater- 
nionic group Sp(n, n) is the group of quaternionic (n + n) x (n + n)-matrices 
preserving the Hermitian form 

n 2n 

{v,w} :=Y^VjWj - V J W 3- 

j=l j=n+l 

We consider its action on Sp(n) by linear fractional transformations (2.19) 
as above. We also consider the representation p\{g) of Sp(n, n) in the space 
C°°(Sp(n)) given by 

Px(g)F(h) = F(h^)dct(a + h 1 y 2n - 1 - x 

Proposition 4.2. a) The operators p\{g) preserve the Hermitian form 

(F 1 ,F 2 ) X = ff txigh-^F^F^KjdfjL^dfiih). 

J JSp(n)xSp(n) 

b) If —n > A > — n — 1, then the Hermitian form (-, -)\ is positive definite 
and thus the representation p\ is unitary. 

The nontrivial part of the statement is positivity of the Hermitian form. 
This follows from Theorem 4.1. 
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5. Unipotent representations. 

Here we discuss models of "unipotent" representations of Sahi and 
Dvorsky-Sahi, 

5.1. The case 0(2n, 2n). In notation of Section 3, we suppose that 

A = — ri + a a = 1, 2, . . . , n. (5-1) 

The first row in (3.3) has zero of order a at our A. The second row in (3.3) 
has a pole of order ^ a. Hence the total expression is nonzero iff the order of 
the pole is precisely a, i.e., 

l n — 0, , ln—1 — 1; ■ • • ^n — a+1 1 

Under this condition, all the coefficients c\ are positive. 

Thus, for A having the form (5.1), the inner product (3.5) is non-negative 
definite (and degenerated) and the operators (3.6) are unitary with respect to 
this inner product. 

Remark. For a = 0, our representation is the one-dimensional representa- 
tion. 

For a = 1, the representation obtained in this way is an element of the 
Molchanov degenerated discrete series, see ^21> now there exists a wide literature 
devoted to this representation, see, for instance |1(J| . 

5.2. Groups U(n, n). Now we assume 

r = 0, fj = — n + a, where a = 1, . . . , n — 1. (5-2) 

The coefficient c m given by (2.10) is non-zero, if m is contained in the union of 
the following disjoint sets Zj, j = 0, 1, . . . , n — a, 

Zg -.m has form (mi, . . . , m n - a -e, " - 1, " - 2, . . . , 0, m n -e+i, ■ ■ .m n ) 

Denote by V m be the U(n) x U(n)-submodule in C°°(U(n)) corresponding 
a signature m, see 2.5. 

Proposition 5.1. The subspace 

W ta u := ®Vf 1 C C°°(U(n)), where /x £ UZ i; 

is a U(n, n) -invariant subspace. 

b) The quotient C°°(X}{n))/Wt a ii is a sum n — a + 1 submodules 

The representation o/U(n,n) in each Vj is unitary. 
A proof is given below. 

5.3. Blow-up. The distribution £ CT)T depends meromorphically in the two 
complex variables a, r. Its poles and zeros are located at a € Z and in t e Z 
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and hence values of this distributions at points (a, r) G Z 2 generally arc not 
uniquely defined. Passing to this point from different directions, we can obtain 
different limits. 

It is convenient to formulate this more formally. 

We fix a point (a, r) = (— n + a, 0) and introduce the new local coordinates 
near this point 4 by 

(T = —n + a + se, t = te (5.3) 
The new coordinates are defined up to the equivalence 

(s,t,s) ~ (su,tu,e/u) (5.4) 

We also think that 

\s\, \t\, \e\ are suffitiently small, and (s,t) ^ (0,0). (5.5) 

If we replace e = 1/R, then the collections (s,t,R) are defined up to the 
equivalence 

(s, t, e) ~ (su, tu, Ru) 

i.e., s, t, u is a point of the projective plane. Thus the set (5.5) can be consired as 
a subset in projective plane. In the new coordinates, the point (— n + a, 0) G C 2 
corresponds to the whole complex projective line (s, t, 0). 

Thus replacing a neibourhood of (— n + a, 0) G C 2 by the set (5.5), we obtain 
a new complex manifold, denote it by C 2 . 

Proposition 5.2. The distribution 

is a meromorphic distribution-valued function on C 2 in the domain (5.5). The 
unique pole (or order n) near (s, t) = (0, 0) is the line s + t = 0. 

Proof. Each Fourier coefficient c m in the formula (2.10) has the form 



- {s + t )— AmM.e) (5-6) 

where the last factor is holomorphic and nonvanishing near the line (s, t, 0), and 
the powers k, j, n — a — j are nonnegative. 

Thus, the Fourier coefficients c m of the distribution ^ s '*' e are meromorphic 
in the our region. 

We must verify that the derivatives d£ s,t ' £ /dt, di s ' t,e /ds, d£ 8,t ' e /de are well- 
defined distributions. 

A central function on U(n) is a distribution, if its Forier coefficient have at 
most polynomial growth in m. For (5.6), this can be verified by a direct tracing. 

4 This construction is the blow-up of the plane C 2 at the point (— n + a, 0). 
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5.4. The family of invariant Hermitian forms. Proof of Proposition 

5.1. Thus, for (a, r) = (— n + a, 0), we have the following family of Hermitian 
forms invariant with respect to the operators (2.22) 



A s ,t(/, g) = / / £ s ' t '°(zu*)f(z)f(u) dy,[z) dv(u) 

J JU(n)xU(n) 

Now we emphasis some additional properties of the formula (5.6). 

First, the exponent k in (5.6) is zero iff m e UZj. Hence Wtail is the common 
kernel of all the forms A s t . This implies the statement a) of Proposition 5.1. 

By the invariance, we subspaces Wj are pairwise orthogonal with respect to 
all the forms A s '*'°. 

Secondly, i? m (s,i,0) is a constant which depend only on m (since noncon- 
stant summands in the linear factors of R m have the form se), te). 

Thirdly, if m G Zj, then the exponent j in (5.6) is our j. This means, that 
the restriction of A s,t '° to Wj has the the form 

p 8 n—a—j 

where S(f,g) is independent of t, s. 

The Fourier coefficients of Zj are the factors R m (s, t, 0) from (5.6), and they 
can easily be written. It is easy to observe that all these coefficients have the 
same sign. 

This finishes proof of Proposition 5.1.b). 

5.5. Expression for the kernel. We can concider the form S 3 as a form 
on C°°(U(n)) extending it to other harmonics as zero. 

Proposition 5.3. The form Sj is defined by the distribution 



1 f)i 

(i+ty-^v-) 



t=o 



1 

lim (i + t)«-«det(l - zu *){"+«+e|te} 
n! e^o dP 



t=0,e=0 



Proof. The first expression is obtained by differention of the block decom- 
position (5.7) of the Hermitian forms A gj t; the right-hand side is a result of 
changing of the limit passing and differentiation. 

Remark. The sum (5.8) contains the well-defined term det(l — zu*)~ n+a := 
£ 1 '°'°(zu*) (that looks like a function) and sum of distributions supported by 
the submanifold rk(l — zu*) = n — 1, n — 2, n ~ a (the analysis of the 
determinantal distributions was done by Ricci, Stein |19j). 
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